Device for creating and/or processing an object signature, monitoring device, method and computer program

ABSTRACT

The invention relates to a device ( 2 ) for creating and/or processing an object signature of an object from a scene, wherein the object signature is designed to describe and/or identify the object, comprising a characteristic extraction unit ( 5 ) designed to extract at least one characteristic value of the object from the scene, and wherein the characteristic value can be displayed in a characteristics space of the characteristic. The device further comprises a coding unit ( 8 ) designed to code the characteristic value into an identification data unit, wherein the identification data unit forms part of the object signature, and wherein the identification data unit refers to a partial region of the characteristic space of the characteristic.

BACKGROUND INFORMATION

The invention relates to a device for creating and/or processing anobject signature of an object in a scene, wherein the object signatureis designed to describe and/or identify the object, comprising acharacteristic extraction unit designed to extract at least onecharacteristic value of the object from the scene, wherein thecharacteristic value can be displayed in a characteristic space of thecharacteristic, and comprising a coding unit designed to code thecharacteristic value into an identification data unit, wherein theidentification data unit forms a part of the object signature. Theinvention furthermore relates to a surveillance device, a method, and arelated computer program.

Known embodiments of video surveillance systems include a plurality ofsurveillance cameras which also observe complex surveillance scenes. Inmany embodiments, the image data streams from the surveillance regionsare combined and evaluated in a central location. In addition to anevaluation being performed by surveillance personnel, automatedsurveillance using image-processing algorithms has since becomeestablished. A typical procedure is to separate moving objects from the(substantially stationary) scene background, track them over time, and,if relevant movements are identified, trigger an alarm or continuetracking the objects.

If the surveillance region covers a wide area, it is also common totrack the moving objects not only within the viewing field of a singlecamera, but also across the entire camera network within thesurveillance region and for an extended period of time. The problemassociated therewith is that of recognizing the moving objects indifferent locations and at different points in time. The reason for thisis that the viewing fields of the cameras often do not overlap, andtherefore a moving object leaves the viewing field of one camera andenters the viewing field of another camera at an undetermined, laterpoint in time. To enable recognition to be performed, as muchinformation as possible is collected about the moving objects when theyappear. This collection of information is called the object signature.Using the object signature, it is now possible to recognize the objectsin the camera network.

Publication DE 10 2005 053 148 A1, for example, which is the closestprior art, discloses a method for handling content information in avideo surveillance system. Accordingly, content information onindividual objects is made available in a network, and is distributedvia the network. The content information is collected at varioushierarchy levels, wherein the various hierarchy levels can be readindependently of each other. It is also provided that the contentinformation is coded in order to compress content information from lowerhierarchy levels into higher hierarchy levels to create overviewinformation, thereby advantageously enabling this overview informationin the higher hierarchy levels to be used in a search for relevantevents, in order to limit a review of image archives of video data toshort, relevant time periods.

DISCLOSURE OF THE INVENTION

A device for creating and/or processing an object signature having thefeatures of claim 1, a surveillance device having the features of claim10, a method having the features of claim 11, and a computer programhaving the features of claim 12 are disclosed. Preferred or advantageousembodiments of the invention result from the dependent claims, thedescription that follows, and the attached figures.

The invention relates to a device that is suitable and/or designed forcreating and/or processing—in particular for the purposes of making acomparison—an object signature of an object in a scene. The object ispreferably embodied as a moving object, such as a person, an automobile,or the like. The scene is understood to be a section of a surveillanceregion that is observed or can be observed using sensors. The sensorsare preferably designed as cameras; as an alternative or in additionthereto, other sensors such as temperature sensors, odor sensors,microphones, etc. can be used. The object signature is a data collectiondesigned to describe and/or identify the object, in particular withinthe scope of recognizing it or performing a comparison.

The device includes a characteristic extraction unit that can extract atleast one characteristic value of the object from the scene, andpreferably a plurality of characteristic values is extracted. Thecharacteristic value can be displayed in a characteristic space of thecharacteristic, the characteristic space being formed by a large numberof possible characteristic values for the characteristic. Everycharacteristic can therefore take on a plurality of characteristicvalues of this type in the characteristic space. For example, thecharacteristic space is designed as a color histogram occupied bystructure characteristics, SIFT characteristics, noises, odors, etc.

Furthermore, a coding device is provided that is designed to code thecharacteristic value into an identification data unit, in particularinto a dimensionless identification data unit, wherein theidentification data unit forms a portion of the object signature.

Within the scope of the invention it is provided that the identificationdata unit refers to a subregion of the characteristic space of thecharacteristic. In particular, the identification data unit for thecharacteristic is not descriptive in and of itself, but ratherrepresents a transformed or mapped value.

In contrast to the known prior art, the characteristic value itself isnot integrated into the object signature. Instead, only a reference tothe subregion that contains the characteristic value is contained in theobject signature. This embodiment has the advantage of ensuring thatobjects are described in a very efficient and compact manner. Instead offorwarding the often highly complex characteristic values, only thereferences—which are comparable to pointers in C syntax—to thesubregions are forwarded with the characteristic values.

According to a preferred embodiment of the invention, the identificationdata units are each designed as a multibit word. Depending on thecomplexity of the characteristic space and the number of subregionsthereof to be taken into account, an 8-bit, 16-bit, or 32-bit word isused as the object data unit, and therefore the storage or transferrequirement for the associated characteristic in the object signature islimited to this word length. The subregion is preferably selected suchthat it combines a plurality of characteristic values. In particular itcan be provided that subregions of a characteristic space comprisedifferent quantities of characteristic values.

Preferably the object signature includes not just one identificationdata unit of a single characteristic, but rather a plurality ofidentification data units that are assigned to various characteristics.The type of characteristics is not limited; for instance, thecharacteristics can describe optical properties such as color,brightness, texture, etc., motion e.g. speed or acceleration, and/oracoustic features e.g. the sound of footsteps, or odors, etc. of theobject. It is basically also possible to use electromagnetic radiationsuch as emission radiation from cellular phones and the like ascharacteristics in the object signature, and to reference them using theidentification data units.

According to a preferred embodiment of the invention, the deviceincludes a subspace generator designed to create the characteristicspace and/or the subregions in the characteristic space. The subregionscan cover the characteristic space entirely, but it is preferable forthe subregions to be separate from each other or to be disposed in thecharacteristic space in a disjoint manner. The characteristic spaceand/or the subregions can be created in a training phase e.g. off-linein advance or on-line during the running time of the device. Preferablycare is taken to ensure that the characteristic space associated witheach characteristic is subdivided into subregions or clusters in ameaningful manner. A meaningful clustering can be attained e.g. byrecording a large quantity of test data (objects that appear in asurveillance network), and to use these test data to occupy thecharacteristic space and subsequently cluster it. The more clusters thatare created in the characteristic space, the finer the differences arethat can be extracted from the objects, but also the more expensive andextensive is the transmission and/or processing of object signaturessince the word length of the identification data unit for thecharacteristic space required for transmission increases as complexityincreases.

Preferably the coding unit is designed such that an object receives theidentification data unit from the subregion that is closest to thecharacteristic value in the characteristic space. The characteristicspace can also be expanded during the running time of the device bydefining new subregions if the characteristic values are too far removedfrom the previous subregions.

In the ideal case it would be sufficient to compare only theidentification data units of the object signatures with each other inorder to detect and identify identical objects. This does function,although typically not in an optimal manner since the characteristicvalues for identical objects vary greatly due to affine distortions,scale changes, lighting changes, localization errors, or in general dueto noise in the characteristic values. For this reason, allegedlyidentical characteristic values (the same characteristic is calculatedfor the same object at another point in time) are not assigned to thesame subregion, and therefore the identification data units for thischaracteristic differ.

Due to this difficulty, it is preferable for the subspace generator tobe designed to create a similarity measure between the subregions of asingle characteristic space. The similarity measure is an assessment ofthe similarity between two subregions. The advantage of introducing thesimilarity measure is that the result of a comparison between an actualobject and a reference object is not dependent exclusively on theidentity of object signatures, but rather on the similarity of objectsignatures.

In one possible embodiment of the invention, the similarity measure isdesigned as a similarity graph, and the subregions are connected asnodes or clusters across paths. The similarity of two subregions isdetermined e.g. by the number of intermediate nodes between twosubregions and/or the length of the paths extending between them,preferably along the shortest path in every case.

According to a development of the invention, the device includes acomparator unit designed to perform a comparison between a first objectsignature having first identification data units and a second objectsignature having second identification data units on the basis of thesimilarity measure. When the comparison is carried out, the similarityfunctions of each characteristic are fused in particular, therebyresulting in a more robust recognition of objects. It lies within thescope of the invention for each of the characteristics to be handledequally; in modified embodiments, the characteristics can also behandled in a weighted manner, and so more significant characteristicsare weighted more heavily in the comparison than are less significantcharacteristics.

A further subject of the invention relates to a surveillance device, inparticular a video surveillance system comprising a plurality ofsurveillance cameras which are and/or may be disposed in a network tomonitor a surveillance region, and comprising at least one deviceaccording to one of the preceding claims, wherein the object signaturesare transferred via the network. In this embodiment of the invention,the advantage of minimizing the data to be transmitted in the case ofthe object signature according to the invention is realized particularlyeffectively.

To enable two object signatures to be compared at any point in thenetwork when one or more data memories are provided in the network, itis preferable to provide one or more data memories in the network, inwhich the characteristic spaces or the subregions and their mapping ontothe identification data, and/or the similarity measure are stored. Whena comparison is performed, after the object signature has beentransferred or generated, the data memory can be accessed from any pointin the network, thereby making it possible to determine the similaritybetween two object signatures.

A further subject of the invention relates to a method for creatingand/or processing and/or searching for or retrieving an object signaturehaving the features of claim 11, the method preferably being implementedon a device of the type or surveillance device according to one of thepreceding claims. The method that has been presented makes it possibleto efficiently store, search for, and/or compare object signatures. Afinal subject matter of the present invention relates to a computerprogram having the features of claim 12.

Further features, advantages, and effects of the invention result fromthe following description of a preferred embodiment of the invention. Inthe figures:

FIG. 1 shows a schematic block diagram of a surveillance device, as anembodiment of the invention;

FIG. 2 shows an illustration of a characteristic space.

FIG. 1 shows a schematic block diagram of a surveillance system 1 whichis suited and/or designed to monitor a complex surveillance region e.g.streets, buildings, factory halls, libraries, etc. Surveillance system 1comprises a plurality of surveillance devices 2 which are connected toeach other via a network 3. Network 3 can have any design e.g. it can bewireless or wired, it can be a LAN, WLAN, the Internet, etc.Surveillance devices 2 each include sensors 4 which detect a portion ofthe surveillance region as a scene. For example, sensors 4 are designedas surveillance cameras. Particularly preferably, surveillance devices 2including sensor 4 are designed as “smart” cameras since the inventionsupports the use of devices that do not have much computing power.

One function of surveillance system 1 is to track moving objects, suchas persons, in the surveillance region. To this end, the moving objectsthat were detected by a first surveillance device 2 must be recognizedin a detection performed by another surveillance device 2. For purposesof recognition, an object signature is created for every detected objectin every surveillance device 2, and is compared with object signaturesfrom other surveillance devices 2 that are distributed throughoutnetwork 3. If the current object signature matches an object signaturefrom another surveillance device 2, then an object is considered to havebeen recognized. If no matching or corresponding object signatures canbe found, then the object is considered to be a newcomer in thesurveillance region. This situation can occur e.g. in entry regions ofthe surveillance region.

To create the object signature, the sensor data from sensors 4, inparticular the images or image sequences from the surveillance cameras,are transmitted to a characteristic extraction unit 5 which detectscharacteristics of a current moving object. For example, it is possiblefor characteristic extraction unit 5 to include a device for objectdetection and tracking 6 which, as mentioned initially, first separatesmoving objects from the substantially static scene background and thentracks them over time. Starting with these detected objects, optical orkinetic characteristics, such as color, brightness, texture, or speed oracceleration, are extracted and a characteristic value is assigned toeach of these characteristics for the current object. Characteristicextraction unit 5 can also contain further modules 7 for extractingother characteristics such as acoustic characteristics, etc.

The characteristic values of the characteristics are transferred to acoding unit 8 which assigns an identification data unit in the form of acluster ID to each characteristic value of a characteristic. Referenceis made to FIG. 2 for an explanation of the cluster ID.

FIG. 2 is a schematicized depiction of a cluster diagram that isintended to define a characteristic space. The characteristic space canbe two-dimensional, as drawn, but can also be three-dimensional ormulti-dimensional. In the characteristic space, potential or detectedcharacteristic values are entered and then combined into clusters 1, 2,. . . 7 e.g. during a learning phase or even during operation ofsurveillance system 1. A meaningful clustering can be attained e.g. byrecording a large quantity of test data and then using these test datato occupy and cluster the characteristic space. The more clusters thatare created in the characteristic space, the finer the differences arethat can be extracted from objects, but likewise the larger the clusterIDs become. The characteristic space can also be expanded during therunning time by forming new clusters. Depending on how many clusters arecalculated in a characteristic space, the size of the cluster ID (8-bit,16-bit, or 32-bit) of a cluster and therefore the memory or transferrequirement for this characteristic type is specified in the objectsignature.

A current characteristic value of a current object 9 is assigned to thecluster that is closest in the characteristic space. For example,characteristic value 9, which is marked with an asterisk, is assigned tocluster ID 1, and characteristic value 10, which is marked with a cross,is assigned to cluster ID 4.

To assign the current characteristic values to the cluster IDs,surveillance device 2 includes a data memory 11 for storing thecharacteristic space and the clusters.

As soon as all characteristic values have been mapped to cluster IDs,the object-signature generation is halted. The object signature is nowcomposed largely or exclusively of the cluster IDs, each of which isdesigned as a multibit word. The majority of information thereforeremains in data memory 11, and only the references to the informationare stored in the object signature. Objects are therefore described in avery efficient and compact manner.

To recognize an object, surveillance device 2 includes a comparator unit12 which compares an object signature, which is transmitted via network3, with the current object signature. Ideally, it would be sufficient tocompare only the cluster IDs of the object signatures with each other inorder to identify objects that are identical or the same. This procedureis typically susceptible to error, however, since the characteristicvalues vary or contain noise. These variations can be created e.g. dueto the object changing its pose and/or to lighting of the objectchanging. As a result of these situation-dependent, static or systematicerrors, the characteristic values of identical objects will bedifferent, and are also likely to be assigned to different cluster IDs.

For this reason, a similarity graph is added to the characteristic spacein FIG. 2, whereby the distance between different cluster IDs isindicated in the form of a path length d or the number of nodes thatmust be passed in the similarity graph, and a measure of the similarityof two clusters is represented. The similarity graph is stored in datamemory 12 or in a central memory in network 3. For the case in which thecharacteristic space and/or the similarity graph changes during therunning time, the changes are forwarded to the other data memory 12 ofthe other surveillance devices 2 or to the central memory or memories.

A similarity comparison between two object signatures is carried out incomparator unit 12 in addition to or instead of an identity comparison.To this end, the similarity of the cluster IDs of two object signaturesis calculated for each characteristic. All similarities of allcharacteristics of the object signatures are then accumulated to obtaina global statement about the similarity of the objects. The accumulationis carried out e.g. by summing, with the characteristics being weighteddifferently. Objects can be recognized in a very robust manner usingthis procedure. The method can be applied to any characteristic, whereinthe subsequent fusion of the similarity claim for each characteristicmakes it possible to detect objects in a robust manner.

When video surveillance is performed using surveillance system 1, it ispossible e.g. to calculate the characteristic values per frame of acamera and assign them to the particular subregion or cluster todetermine the cluster ID. In order to stabilize the object signature,only those cluster IDs are used that are determined often i.e. in aplurality of frames.

1. A device (2) for creating and/or processing an object signature of anobject in a scene, wherein the object signature is designed to describeand/or identify the object, comprising a characteristic extraction unit(5) designed to extract at least one characteristic value of the objectfrom the scene, and wherein the characteristic value can be displayed ina characteristic space of the characteristic, and comprising a codingunit (8) designed to code the characteristic value into anidentification data unit, wherein the identification data unit formspart of the object signature, characterized in that the identificationdata unit refers to a subregion of the characteristic space of thecharacteristic.
 2. The device (2) according to claim 1, characterized inthat each identification data unit is designed as a multibit word and/orat least one subregion comprises a plurality of characteristic valuesand/or a plurality of subregions of a characteristic space comprises adifferent number of characteristic values.
 3. The device (2) accordingto claim 1, characterized in that the object signature comprises aplurality of identification data units which are assigned to variouscharacteristics.
 4. The device (2) according to claim 1, characterizedby a subspace generator designed to create the characteristic spaceand/or the subregions in the characteristic space.
 5. The device (2)according to claim 4, characterized in that the subspace generator isdesigned to occupy and/or update the characteristic space and/or thesubregions of the characteristic space on the basis of test data and/orreal-time data.
 6. The device (2) according to claim 4, characterized inthat the subspace generator is designed to create a similarity measurebetween the subregions of a single characteristic space.
 7. The device(2) according to claim 6, characterized in that the similarity measureis designed as a similarity graph, and the subregions are connected asnodes across paths.
 8. The device (2) according to claim 7,characterized in that the similarity between two subregions isdetermined by the number of intermediate nodes and/or the length of theintermediate paths.
 9. The device (2) according to claim 1,characterized by a comparator unit (12) designed to perform a comparisonbetween a first object signature having first identification data unitsand a second object signature having second identification data units onthe basis of the similarity measure.
 10. A surveillance device (1)comprising a plurality of surveillance sensors (4), in particularcameras, which are and/or may be disposed in a network (3) to monitor asurveillance region, characterized by a device (2) according to claim 1,wherein the object signatures are transferred and/or can be transferredvia the network (3).
 11. A method for creating and/or processing and/orsearching for an object signature of an object in a scene, wherein theobject signature is designed to describe and/or identify the object,preferably using the device (2) and/or the surveillance device (1)according to claim 1, at least one characteristic value of the object isextracted from the scene, the characteristic value being taken from acharacteristic space of the characteristic, and wherein thecharacteristic is coded into an identification data unit, theidentification data unit forming part of the object signature, whereinthe identification data unit refers to a subregion of the characteristicspace of the characteristic.
 12. A computer program comprising programcode means for carrying out all steps of the method according to claim11 when the program is run on a computer and/or a device (2) and/or asurveillance system (1).